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Projects of the National Center for Higher Education 
Hanu ' ^ent Systeas (^CBEBS) conducted to dereloi: techniques and 
strategies fcr describing and analyzing oa-^soaes are discussed. It is 
suggested that the inforaation vill assist decision^aakers in 
anderstanding and using otttccaes inforaation in their planning a^id 
■anageaent decisions. The Center's orientation to the outcoaes 
iafortation problaa and the developaent o£ their research design are 
explained. The obiectives of each outcoaes-related activity and the 
products developed through these activities are described. Halor 
goals of the oatcoaes pro-fects include: (1) identifying indicators of 
postsecondary education oatcoses: (21 defining each indicator in a 
standard fashion: (3) develop^ing procedures for acquiring data 
relative to each indicator: and dissaainating the inforaation and 
products to decision -Bakers in the postsecondary ccBBUnlty. Ten 
oatcoaes- re2 a ted projects undertaken by NCfiENS are briefly 
susaarizedf vith their potential t^pplications and" basic source 
docuaents listed. (SF) 
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NCHEMS OUTCOMES-RELATED ACTIVITIES AND PRODUCTS: AN OVERVIEW 



Introduction 

As its name Irapliec, the National Center for Higher Education Management 
Systems (NCHEMS) is concerned with developing planning and management systems 
for Institutions and agencies of postsecondary education. Over 650 institutions 
and agencies of postsecondary education currently support and participate in 
the NCKEMS program. Constituent involvement always has been a vital part 
of the underlying philosophy of the Center. From Its earliest inception in 
1965, when it was first suggested that interstate cooperation in the application 
of management science in postsecondary education should be investigated, those 
likely to be affected by the Center's work have been actively involved in 
the NCHEMS program. These institutions and agencies include all sectors of 
the postsecondary education community: postsecondary education associations, 
national professional organizations, regional compacts for education, state- 
wide coordinating asd governing boards, community and junior colleges, multi- 
campus systems, and private and public colleges and universities from across 
the nation. It is estimated that the institutions participating in the NCHEMS 
program enroll over eight million students. 

Although NCHEMS always has recognized that both outcome infcrmation and cost 
information are necessary for effective planning and management in*post- 
secondary education, NCHEMS did not Immediately attack both problems on the 
same scale. 
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The Center's initisl efforts focused primarily on the developMnt of 
procedures for TOasuring, analyzing, and communicating institutional 
and program costs. This strategy was not only a result of the size of 
the problem and the inherent resource limitations but also a response 
to the needs and preferences of NCHEMS' constituents. 

However, as pressures for accountability increased and the postsecondary 
educaton community began feeling the financial pressures of rising 
costb and limited resources, NCHEMS recognized that it no longer could 
focus primarily on the development of cost information— the problem 
of developing outcome information had to be addressed also. As a result, 
a series of projects have been conducted to develop techniques and strategies 
for describing and analyzing educational outcomes and to assist decision 
makers in understanding and using outcome information in treir planning and 
management decisions. This paper briefly describes the objectives of the 
Center's outcomes -re la ted activities and discusses the products developed 
through these activities. 



Long-Range Approach 

At the beginning of the Center's work in the outcomes area, the major objective 
was to become more familiar with the outcomes measurement and anal /sis problem 
in higher educaton and determine how NCHEMS could aid its. constituents in 
dealing with the problem. Pursuing this objective, the staff began studying 
what had been done in this area and examining the questions decision makers 
were asking relative to higher education outcpmes. Many discussions were 





held with practitioners, researchers., and others Interested in the outcomes 

« 

information problem. A national seminar was held in Washington D.C. in 1970 
for the purpose of highlighting hte 'problems of Identifying, measuring, 
and analyzing the outc«nes of higher education. A number of authorities 
in the field were brought 'together to present position papers on the 
subject and the results of the seminar were crmpiled and published (Lawrence, 
et al., 1970). Another major activity during this early stage was the 
establishment of a Design Ccmtnittee for the purpose of conceptualizing and 
stimulating thinking about what research development, and implementation 
efforts should be undertaken. ^ 

As the project work progressed, this early .objective gave rise to an overall 
conception of the outcomes area that could serve as an operational basis 
for future work. The outcomes -re la ted work currently unde^^ay at NCHEMS 
rests on the premise that in order to collect and use 1nfonr.at1on about the 
outcomes of postsecondary education, three related needs must be filled: 

1. The need to provide a comprehensive picture of 3 outcomes 
of postsecondary education and to develop the capability to 
measure these outcomes. 

2. The need to provide a structure for organizing outcomes informa- 
tion as a prerequisite for the analysis and communication of 
this Information. 

3. The need to develop analytic procedures to apply this informa- 
: tion to the solution of particular planning and management 

problems. 

Major components of the outcomes-related projects are directed toward each 
of these needs. 

With regard to measures of educational outcomes, NCHEMS staff are working 
to: (1) identify indicators of postsecondary education outcomes, (2) define 



each of these Indicators. In a standard fashion, (3) develop procedures for 
acquiring data relative to each^of these Indicators, and (4) disseminate the 
information and products to decision makers in the post secondary education 
coBinunity* ' . . 

Two- major criteria continue to be considered In the development of the full 
range of postsecondary education outcrane indicators. First, they should 
reflect a recognition that information needs vary considerably across different 
decision makers. Second, the measures should acknowledge that educational 
outcomes occur over an extended period of time. 

ti J 

In responding to the need for an organizing structure,^ the staff ar^ 
working t j: (1) develop a general categorization scheme for classifying and 
organizing outcome measures, (2) identify the distinctly different types of 
educational outcomes , (3) categorize measures in accordance with these dis- 
tinctions, and (4j provide procedures whereby measures can be arrayed within 
these major categories in a logical and useful fashion. In developing the 
structure , attention is given to ensure that it not only accommodates the 
full range of outcomes but also makes allowances for the various outcome 
dimensions such as long-term vs. short-term and the multiple levels of detail-- 
from individual to society— that must be accommodated. 

In a^iition to organizing measures for ease of communication, this structure 
should provide a focal pg^t for conceptual and theoretical discussions 
surrounding the question, "Just what are the outcomes of postsecondary 
education?" The development of a comprehensive outcomes structure requires 
the resolution of the very difficult question of what should be considered 
an outcome of postsecondary education and what should not. The structure 
will serve also to identify categories of outcomes for which measures or 



indicators are not yet identified. Used In this way, a fonral structure can 

i 

not only provide a mechanism to assist in the coordination of research and 
development efforts but also serve as a device to highlight those outcoines 
for which quantitative measures are not available and thus reinforce the need 
to deal with such outcomes qualitatively. 

« 

Inasmuch as the ultimate objective is improved planning, management, and 
evaluation, NCHEMS Is placing particular en^hasis on the identification and 
development of relevant analytic capabilities that will foster the interpre- 
tation and use of outcome data. In particular, attention is being focused 
on- developing capabilities for investigating relationships among input or 
process variables and resulting outcomes. It is anticipated that users will 
benefit not only from the actual findings of these analyses in case study 
forms but also from the dissemination of prototype analytic procedures and ^ 
techniques applicable in undertaking analyses of particular kinds of problems. 
In carrying out these tasks, NCHEMS is relying heavily on a strategy of 
producing prelimi/iary products early In the development cycle and then 
concentrating on making improvements wherever possible. This development 
approach is also characterized by the active involvement of the potential users 
of the project results at each major stage. • NCHEMS has consistently favored 
a style of opera ti on that solicits input from prori'jct users during the develop- 
ment proress. Historically, this invoivemer.t has been obtained through task 
forces, advisory p roups, fi^^[ reviews and pilot tests. In the outcomes- 
related projects, these iri^olvement mechanis.Tis have been augmented by surveys 
and other devi ces that allow the staff to acquire user input in a more systematic 
and specific way. 



NCHEMS Outcomes Products in Context 



Out of the initial work of the NCHEMS Outcdaies of Postsecondary Education Project . 
came a document (Micek and Wallhaus, 1973) that presented a "Framework for / 
Understanding the Use of Higher Education Outcome Information" aifd an 
"Inventory of Higher Education Outcome Variables and Measures." Nine addi- 
tional major sets of outcomes products have since been developed or are 
currently under development. Each will be described here briefly, in chrono- 
logical order of development. Their potential applications and basic source 
documents are listed in Figure 2 on pages 10-11 (in alphabetical order by name). 
Taken as an arrayfthese sets of products also suggest the importance of not 
considering educational outcomes in isolation, 

• Inventory of Higher Education Outcrme Variables and Measures . This 

is a logically organized listing of outconre categories apd subcategories 
with a definition and a list of potential measures and indicators for 
each. The overall broad categories are three in number: (1) Student 
Growth and Development, (2) Developnwnt of New Knowledge and Art Forms, 
and (3) Conmunity Development and Service. In addition, formal pro- 
cedures are provided for using the Inventory in translating goals into 
measurable terms and developing a priority list of outcomes. 

• Inventory of Instit ution al Environm en t Variables and ^teas^res . This is 
a list of insti tutionaTenvironment categories and subcategories with 

a set of suggested indicators or measures associated with each. It was 
constructed under the rationale that many peoeple view the educational 
environment as an educational outcome and that outcomes must be descri bed 
and understood in relation to the envi ronment within which they occur. 
The five major categories in this inventory are (1) Listrurtional /Research 
Environment, (2) Physical Environment, (3) Organizational Environment, 
(4) Social Environment, and (5) Economic -Environment. 

t Outcome Measures Indentlfi cation Study . One way to generate goals is to 
ask relevant higher education decisionmakers what outcome measures and 
indicators will be helpful to them in their day-to-day decisionmaking, 
and then inductively arriving at the goals that they imply. This study 
involved asking five groups to respond to a list of outcome measures in 
tertns of (1) the extent to which they needed the measure for their 
decisionmaking, and (2) the access they had to data for the measure. 
The groups surveyed were institutional academic vice presidents, 
institutional student affairs vice presidents, institutional finance 
presidents, state-level higher education 'administrators, and state 
legislators. The questionnaire and procedures used have served as 
a model for a number of similar studies within insti tutiona. 
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• Outcome ^asures and Prt>cedures Manual . This manual presents suggested 
standard definitions, data sources, and data acquisition procedures for 
over forty outcome roeasur-es foun<J ^o have high priority In the Outcome 
Measures Identification Study discussed above . An expanded version of 
this manual Is currently under development. " 

« 

• Student Outcomes Questlcmnaires , Student outcomes probably have the 
highesjt priority at most colleges, and to really understnad them requires 
a longitudinal measurement approach. Furthermore, a longitudinal 
approach encourages better planning for the assessment of student 

, outcomes, results In Increased ability to follow students, means 
less duplication and, shorter questionnaires for students, increases 
the probability that the needed outcomes Information will be collected 
at the proper time, and allows, the questlonnarie administration to be 
integrated Into the systematic, ongoing process of the institution. 

This product group, which has been developefl jointly by NCHEMS and 
the College Board, inc^.udes a series of tried and tested basic student- 
outcomes questionnaires (entering student, current student, dropouts , 
graduating student , and recent alumni) for two-year colleges and four- 
year colleges and universities, a manual for using the questionnaires 
and the data collected, a questionnaire response-analysis service, 
e;nd comparfati ve data. 

• Outcomes Structure . This is a framework and system for organizing the 
full range of ptistsecondary education outcomes and .Information about 
those outcomes for purposes of classification, -nalysis, and decision- 
making. Similar in concept to the outcomes Inventory described 
previously. It is more comprehensive in its covet^ge , more neutral 

in its orlentati on , more flexible in its application, and it includes 
three dimensions of outcori^s rather than one. The three dimensions 
are "audience" (who or what is intended to, or actually does, receive 
or be affected by the outcome), "type-of -outcome" (the basic entity 
on which the outcome is focused, and whether it involves maintenance 
or change), apd "time" (when the outc'ome occurs or is expected to occur, 
and its actual or expected duwtion) . Heirarchies of categories, 
along with standard definitions and illustrative meaiures and indi- 
cators for each' category, are presented separately for the dimensions. * 
Procedures for practical application of the structure to meet particular 
needs of institutional personnel are outlined. 

• Instit utional Goal Achievement: Measures of Progress . The widely used 
Institutional Goals Inventory of Educational Testing Service lists a 
number of specific goal statements for each of twenty .goal 4 areas, and 
asks respondent, to rate the importance of the statements in terms of 
"should be" and "is.". For each of these same 20 goal areas, this new 
NCHEMS instrument lists" a numbtr of potential measures of progress. 
Respondents are asked to rate the importance of each area as a goal, 
and how appropriate each measures is for measuring progress in achieving 
that goal, at their Institution. 

f 

• i^ommunity In>pact Assessment . Every postsecondary education Institution 
has various kinds of Impacts on the local conmunity, and perhaps on other 
conmunl ties— for example, economic impacts, educational irppacts, social/ 
cultural Impacts, recreational impacts, and technological impacts. Being 
able to demonstrate and document such impacts can have major consequences 



for the economtc and other types of support that an institution receives, 
and this is^ especialljf true for coomuni ty-based institutions. Heretofore, 
roost conwunity ijnpact stu4ies have neglected the important inpacts that 
are outside of the economic area. Turthermore, they have often not 
looked spearately at the inpacts' on Important coiranunity subgroups, and 
have tended to overlook the indirect impacts through faculty, staff, 
students, and graduates of the institution, THis product includes 
practical procedures and guidelines for conducting, and using the results 
from, a coa^rehensive community impact study «at two-year community 
colleges. 

• Needs Assessment Framework . Before one develops institutional goals and 
plans for achieving those goals, it iS essential to determine constituent 
needs and relate them to institutional mission. The same is true at the 
program level. Yet what usually happens is that constituent needs are 
aissessed subjectively in an "off tiae top of the head" manner, ffodels are 
lacking for formal needs assessment in higher education, -nd the ones 
available lack important characteristics such as the following: (1 ) is 
systematic; (2) is coraprehensivei (3) gathers input from many relevant 
groups; (4) derives and ranks both "what should be" and "what is;" 

(5) emphasizes understanding as well as identification; (6) distinguishes 
between needs, wants, and demands ; (7) relates client needs to in- 
stitutional considerations; and (8) considers information In interaction 
rather than isolation. This product includes a conceptual framework and 
general<»altemative procedures for conducting needs assessment studies- 
in postsecondary educatic^. 

• C ompendium of Research Related to Selected Outcome Measures. Separately 
for selected measures (and particularly those Included in the Outcome 
MeasurcB and Procedux^ea Manual} In selected categories of the NCHEMS 
Outcomes Structure, this product provides a sny thesis of the research 
literature that speaks to the validity, reliability and applicability 

of each measure. Annotations of selected research studies are also *^ 
provided for each measure. 
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Figure k 
NCHEMS OUTCOMES-REUTED PRODUCTS 



PRODUCT SET t> 


POTENTIAL APPLICATIONS 


BASIC RESOURCE DOCUMENTS 


' COMMUNITY IMPACT ASSESSMENT 


• Identification of constituent needs 

• Assessment of constituent needs 

• ideeitffica Ion of comnuhity Impact measures and 
indicators 

• Collection of data on comnunity,. impact 
tCNifnunicaticn aoout community impacts with 
constituents and publics 


Conductino Cofrmu^i ty-^p'act 
Studies: A Handbook ff?V« Comfnunity 
Colleqes, 1978 

Assessinq Cbmmunity-Colleae 
Impacts: Three Case Studies, 1979 


INSTITUTIONAL GOAL ACHIEVE. :ENT: 
MEASURES OF PROGRESS (IGA:MP) 


• Translation of goals to measureable objectives 

• Identification of^ outcome "leasures and indica- 
tors 

• Outcomes data collection 

• ^fvaluation 


Measures of Institutional 
Goal Achievement, ,1P78 

^^IH: 


INVENTORY OF KIGHER EDUCATION 
OUTCOME VARIABLES AND MEASURES 

• 


• Stimulation of thinking about outcomes ,in a 
concrete and systematic manner 

• Determination of outconie. goals and priorities^ 

• Translation of goals to*measurable objectives^ 

• Identification of outcome measures and indica- 
tors 

• Assessing alternative* cdyrses of action in 
terms of outcomes 

> cvaiuatlor * 

• Communication about outcomes . 


An Introduction to the Identifica- 


tion and Uses of Higher Education 
Outcome Infonriatlon, Technical 
Report 40, 1973 , 


INVENTORY OF INSTITUTIONAL 
ENVIRONMENT VARIABLES AND 
MEASURES 

« 


• S'-.imulatfon of thinking about environment 

• Dptn^rmi na nn of n>*nrpcc Vinal c 

IpiSI III 1 flQ w 1 Uli W 1 pf WWC«d9 19. , 

• Translation of goals to measurable objectives 

• Assessing alternative courses of action in 
terms of environment 


"Inventory of Institutional 

Lnv 1 runiuEnt variaoica anu ncaburcS 

unpublished document, 1974 


. NEEDS ASSESSMENT FRAMEWORK 


• Identification of constituent needs 

• Assessment of constituent needs 


A document that synthesizes the 
needs assessment literature is in 
preparation 



PRODUCT SET 


POTENTIAL APPLICATIONS 


BASIC RESOURCE DOCUMENTS 


OUTCOME MEASURES IDENTIFICATION 
STUDY 


• Identification of priority outcoire measures and 

i ndi ra'fni''S -x. 

1 liW 1 w \A Wi 9 

• Stimulate thinking about outcome me&sures and 
indicators that are needed 

• Determination of outcome information accessi- 
bility 


'he Higher Education Outcome 

rieosUrea iUenUT iCuulun OT>UQy< 

A Descriptive Summary, T X 


HIGHER EDUCATION OUTCOME 
MEASURES AND PROCEDURES 


• Stimulation of thinking about outcomes In a 
concrete and svsteinatic manner 

• Identification of outcome measures and 
indicators 

• Collection of outcomes data 

• Communication about outcomes 


Outcome Measures and Procedures 


Manual Tor'hn4rrji1 Donni*^ 7n 1 07C 

(This manual is currently 
being revised.) 


OUTCOMES STRUCTURE 

t 


• Stimulation of thinking about ou*-comes in a 
concrete and systematic manner 

• Developing needs assessment studios 

• Developing needs assessment questionnaire 
i terns 

• Determination of outcome goals and priorities 

• Assessment of goals adequacy 

• Translation of goals to measurable objectives 

• Identification of outcome measures and 

indicators 

• Assessino alternative couraes of action in 
terms of outcomes 

• Evaluation 

• Communication about outcomes 

• Outcomes research 


A Structure for the Outcomes of 
Postsecondary Education, 
1577 

An Overview of the Outcomes 
Structure, and Its Application in 


Postsecondary Education Institu- 
tions, 1977 

Previous Attenjpts to "Structure" 


Educational Outcomes and Outcome- 
and Review of the literature, 1977 


STUDENT OUTCOME QUESTIONNAIRES 
AND RELATED SERVICES 
(Products and services jointly 
developed by NCHEMS and the 
College Board and dissefninated 
through the NCHEMS-Coflege Board 
Student Outcomes Information 
Services (SO IS) 


• Stimulation of thinking about student outcomes 
and their assessment 

• Identification of student oi tcome measures and 
indicators 

• Collection of outcomes data 

• Evaluation 

• Communication about outcojnes 

• Development of a longitudinal student data base 

• Response-analysis service 

• Comparative data 

— .1,1,-.. 


Student Outcomes Ouestlonnaires: 
An Imolementatior. H^indbook, 1979 

Student Outcomes Questionnaires: 
Community College Series, 

Student Outcomes Questionnaires: 
Four-Year Colleoe and Universitv 
Series, 1Q7Q 

A Manual for Conductina Student 
Attrition Studies (Revised "1979) 



PRODUCT SET 


POTENTIAL APPLICATIONS 


BASIC RESOURCE DOCUMENTS 


COMPENDIUM OF RESEARCH RELATED 
TO SELECTED OUTCOME MEASURES 


• Stimulation of thinking about outcoines 
assessment 

• Outcome measures and Indicators development, 
modification, and/or evaluation 

• Identification of outcome measures and 
indicators 

• Outcomes research 


A data bank and summary document 
are in preparation 

0 



